Introduce benchmark tests #4105
Conversation
Codecov Report
@@ Coverage Diff @@
## main #4105 +/- ##
==========================================
- Coverage 92.72% 90.52% -2.20%
==========================================
Files 298 159 -139
Lines 8297 3757 -4540
Branches 1728 835 -893
==========================================
- Hits 7693 3401 -4292
+ Misses 604 356 -248 |
There was a problem hiding this comment.
@martinkuba, thanks for working on this, I think this a much-needed addition to the project 🙂
Non blocking: I'm guessing ts-node cpu/memory overhead is why the tests are in plain JavaScript, right? As the current state requires us to run the compile script for the package we're benchmarking anyway, would it make sense to write the benchmark code in typescript, compile it, and then run the compiled benchmark instead? 🤔
|
@pichlermarc Thanks for the review. Yes, I did not want to use |
|
@pichlermarc @dyladan I know you have both already approved this (thank you). Per Marc's comment, I have drafted an alternative in Typescript. Please let me know which version you prefer. |
Which problem is this PR solving?
This project used to have a few benchmark tests in the past (see this past work). However, I think they were removed because they were not very useful. With this PR, I am re-introducing performance benchmark tests with the goal of eventually having them automated and consistent with published results.
The Performance SIG that is forming will work on a standard way of running these tests across various languages and publishing the results in a consistent way.
Related issue: #3940
Short description of the changes
This PR adds a single test based on what is described in this spec. This is a first step. A follow-up to this PR will be a workflow that will automatically run these tests and publish the results.
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
npm run test:benchChecklist: